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ABSTRACT 

Systems engineering is the application of mathematical and scientific principles to practical ends 
in the life-cycle of a system. A methodology for systems engineering is a carefully developed, relatively 
complex procedure or process for applying these mathematical and scientific principles. There are many 
systems engineering methodologies [or possibly many versions of a few methodologies] currently in use 
in government and industry. These methodologies are usually tailored to meet the needs of a particular 
organization. It has been observed, however, that many technical and non-technical problems arise 
when inadequate systems engineering methodologies are applied by organizations to their systems 
development projects. This paper discusses various criteria for evaluating systems engineering meth- 
odologies. Such criteria are developed to assist methodology-users in identifying and selecting method- 
ologies thqt best fit the needs of the organization. 
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SUGGESTED CRITERIA FOR EVALUATING SYSTEMS 
ENGINEERING METHODOLOGIES 


Introduction 


This paper is one of several results of a Dynamic Systems Engineering Methodology Research 
Study being conducted at Howard University under a grant from NASA. The study is sponsored by the 
Networks Division of the Mission Operations and Data Systems Directorate (MO&DSD) at Goddard 
Space Eight Center. The objective of the study is to examine systems engineering methodologies in 
light of changing environments and changing needs. The results of this investigation are to be used to 
identify and validate new methodologies with potential applications to NASA’s systems life-cycle 
processes. 

The study is divided into two phases. Phase One is a study of NASA’s projects, its organization, 
resources, and environment to identify factors that affect the successful application of systems engi- 
neering methodologies. Phase Two involves evaluating existing methodologies, tools, and techniques 
with potential application to NASA’s systems project. 

The criteria for evaluating systems engineering methodologies were developed based on the 
findings in Phase One. These criteria are to be used as a guide and weighing scale for evaluating existing 
systems engineering methodologies in Phase Two of the project, and in making recommendations to 
NASA 


Purpose of a Systems Engineering Methodology 

Systems engineering as described by Blanchard is a process employed in the evolution of system’s 
development from the time when a need is identified through production and/or construction to the 
ultimate deployment of that system[l, p. 1 1]. The series of steps involved in this process is a systems 
engineering methodology. A methodology is primarily used to improve the effectiveness of the overall 
system. It provides a means to increase reliability, decrease downtime, maintain cost effectiveness, and 
avoid redundant and wasted efforts. Furthermore, it provides a means of checking, cross checking, and 
quality control. 

A systems engineering methodology is almost vital for large-scale projects because the success of 
such projects depends upon a strong systems approach to integrate diverse elements into a harmonious 
whole[2, p. 2]. It is also vital to proper project management. The steps of a systems engineering 
methodology are generally presented in the context of a system life-cycle. A life-cycle is a logical 
evolutionaiy flow of what has to be done for the duration of the project. 
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Participants in the Systems Engineering Process 

There are two basic participants involved in the systems engineering process — the problem 
originator and problem solver. The problem originator is the individual who has a problem that needs 
to be solved. Typically, the problem originator is referred to as the client, decision-maker, manager, 
sponsor, or problem owner[3]. The problem solver is responsible for providing the problem originator 
with a solution to his problem. This role is likely to be filled by a person called a consultant, analyst, or 
designer. The problem solver is also the methodology-user for (s)he is the one who uses the set of 
procedures, which may or may not be formally defined, to create an environment whereby a solution 
can be brought about[3]. Therefore, it is essential that the user identifies what tasks must be carried out 
to obtain the desired results. These tasks and the persons responsible for completing each task must be 
clearly defined in the organization’s methodology. 


Suggested Criteria for Evaluating Systems Engineering Methodologies 

The criteria that were developed to evaluate a systems engineering methodology fall within five 
major categories: Structure, Flexibility, Accountability, Documentation, and Special Considerations of 
User (in this case NASA). Structure addresses the composition of the system life cycle process and its 
ability to accommodate simple to large-scale systems. Flexibility refers to the methodology’s ability to 
adapt to change. The third area of interest is accountability. Accountability addresses the ability of the 
systems engineering methodology to ensure that proper procedures are being applied as intended and 
that appropriate procedures are being kept. Documentation refers to how well the methodology is 
written; the level of detail, clarity and ease with which it can be followed. Lastly, because all methodol- 
ogies are tailored to meet an organization’s particular needs, it is necessary to examine the factors that 
are of critical concern to that organization (in this case NASA). The criteria that were developed for 
each category are as follows: 


1. Structure 

• Does the methodology address activities that are likely to involve engineering work such as 
design, construction, installation, and operation? 

• Is it structured to handle large-scale or complex systems (interacting components to achieve 
defined objectives)? 

• Is it structured to handle at least one component that is extensively hardware? 

• Is the methodology partitioned into clearly defined and logical phases, processes, activities, 
or tasks that can be used as a basis for resource allocation and events such as the start or 
completion of phases that can be used as milestones or decision points? 
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2. Flexibility 

• Does the methodology accommodate systems of varying size, nature 
(such as utility/public sector, military, consumer products), and complexity? 

• Does the methodology address ways of handling new information, feedback, 
or unforeseen circumstances (such as new requirements)? 

• Does the methodology allow for acquisition through a variety of approaches 
(procurement, development, etc.)? 

• Does the methodology allow maximum flexibility with time-allocation (scheduling) 
of resources? 

• Does the methodology address ways of identifying and selecting the best human 
and material resources to assign or allocate to its various phases? 


3. Accountability 

• Does the methodology specify the documentation that is appropriate at different 
points during its application? 

• Does the methodology provide for communication and information exchange to 
ensure that all participants are aware of significant project decisions and have the 
most up-to-date information on the project status and activities? 

• Does the methodology specify an auditing or tracking procedure to ensure that it 
has been applied as intended? 

• Does the methodology suggest a management structure to ensure that it is applied 
as intended? 

• Does the methodology identify its intended users, class of systems, and scope of its 
intended applications? 

• Does the methodology provide ways of addressing critical considerations such as 
national security, risk (environmental, evolving technologies), human safety, etc.? 


4. Documentation 

• Is the methodology written clearly, precisely, completely, and at a level of detail 
that is appropriate for its intended users? Is it a good road map? 
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5. Special Considerations of NASA 

• Is the methodology fairly independent of organizational structure? 

• Does the methodology allow for the retention of key personnel throughout the life-cycle? 

• Does the methodology provide for the incorporation of requirements identified during 
the system analysis, design, or subsequent phases? 

• Does the methodology provide tools and techniques for predicting or projecting future 
requirements, through the planning horizon? 

• Does the methodology suggest strategies and techniques for designing and developing 
systems in the absence of specific requirements? 

• Does the methodology provide tools and techniques (including graphics and prototyping) 
for communicating among individuals and various organizations or organizational units 
working on major systems projects? 

• Does the methodology provide tools and techniques for redesigning and making major 
modifications to extend the useful life of a system in operation? 


Summary 

As mentioned, these criteria will be used in Phase Two of the project to evaluate other agencies’ 
and authors’ systems engineering methodologies with potential applications to NASA- They will serve 
as a guide and weighing mechanism for justifying our recommendation to NASA/Goddard. 
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